Robust Adaptive Data Encoding and Restoration 

NCC-1-258 


Summary of Research 
submitted to 

NASA Langley Research Center 
Hampton, Virginia 23681 


Principal Investigator 

Stephen K. Park 
Professor and Chair 

Co-Principal Investigator 

Zia-ur Rahman 
Research Assistant Professor 

Co-Principal Investigator 
Nesim Halyo 
Research Professor 

Department of Computer Science 
College of William Mary 
PO Box 8795 

Williamsburg, VA 23187-8795 
ATTN: Grants &: Research Administration 


June 9, 2000 



This is the final report for NASA cooperative agreement NCC-1-258 and covers the period from 
01 October, 1997 to 11 April, 2000. Enclosed are copies of the research products for which the 
research was supported by the grant, at least partially. 

The research during this period was performed in three primary, but related, areas. 

1. Evaluation of integrated information adaptive imaging. 

2. Improvements in memory utilization and performance of the multiscale retinex with color 
restoration (MSRCR). 

3. Commencement of a theoretical study to evaluate the non-linear retinex image enhancement 
technique. 

The research resulted in several publications, and an intellectual property disclosure to the NASA 
patent council in May, 1999. The publications are listed chronologically in each category. 


1 Information Adaptive Imaging 

1. We have previously published extensively on the information theoretic assessment of imaging 
systems. In this paper, this concept is extended to include the effects of the display device, 
and the interactions with the human observer. 

F. O. Huck, C. L. Fales, R. Alter- Gartenberg, S. K. Park and Z. Rahman, “Information- 
theoretic assessment of sampled imaging systems,” Optical Engineering, Vol. 38, No. 5, 
pp. 742-762, (May 1999). 

Abstract: By rigorously extending modern communication theory to the assessment of sam- 
pled imaging systems, we develop the formulations that are required to optimize the perfor- 
mance of these systems within the critical constraints of image gathering, data transmission, 
and image display. The goal of this optimization is to produce images with the best possible 
visual quality for the wide range of statistical properties of the radiance filed of natural scenes 
that one normally encounters. Extensive computational results are presented to assess the 
performance of sampled imaging systems in terms of information rate, theoretical minimum 
data rate, and fidelity. Comparisons of this assessment with perceptual and measurable per- 
formance demonstrate that (1) the information rate that a sampled imaging system conveys 
from the captured radiance field to the observer is closely correlated with the fidelity, sharp- 
ness and clarity with which the observed images can be restored , and (2) the associated 
theoretical minimum data rate is closely correlated with the lowest data rate with which the 
acquired signal can be encoded for efficient transmission. 

2. The concept of end-to-end imaging and its assessment is well described in the literature. 
However, simulations required to reproduce the reported results can be difficult to design 
without a proper understanding of the representations of the functions in the spatial and 
spatial frequency domain. This paper lays out in detail these various relationships and how 
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they can be used to properly simulate a sampled imaging system, and, hence, analyze its 
performance. 

S. K. Park and Z. Rahman, “Fidelity analysis of sampled imaging systems, ” Optical Engi- 
neering, Vol. 38, No. 5, pp. 786-800, (May 1999). 

Abstract: Many modeling, simulation and performance analysis studies of sampled imaging 
systems are inherently incomplete because they are conditioned on a discrete-input, discrete- 
output model that only accounts for blurring during image acquisition and additive noise. 
For those sampled imaging systems where the effects of digital image acquisition, digital 
filtering and reconstruction are significant, the modeling, simulation and performance analysis 
should be based on a more comprehensive continuous-input, discrete-processing, continuous- 
output end-to-end model. This more comprehensive model should properly account for the 
low-pass filtering effects of image acquisition prior to sampling, the potentially important 
noise-like effects of the aliasing caused by sampling, additive noise due to device electronics 
and quantization, the generally high-boost filtering effects of digital processing, and the low- 
pass filtering effects of image reconstruction. Yet this model should not be so complex as 
to preclude significant mathematical analysis, particularly the mean-square (fidelity) type of 
analysis so common in linear system theory. In this paper we demonstrate that, although the 
mathematics of such a model is more complex, the increase in complexity is not so great so 
as to prevent a complete fidelity- metric analysis at both the component level and at the end- 
to-end system level. That is, computable mean-square-based fidelity metrics are developed 
by which both component-level and system-level performance can be quantified. In addition, 
we demonstrate that system performance can be assessed qualitatively by visualizing the 
output image as the sum of three component images, each of which relates to a corresponding 
fidelity metric. The cascaded, or filtered, component accounts for the end-to-end system 
filtering of image acquisition, digital processing, and image reconstruction; the random noise 
component accounts for additive random noise, modulated by digital processing and image 
reconstruction filtering; and, the aliased noise component accounts for the frequency folding 
effect of sampling, modulated by digital processing and image reconstruction filtering. 


2 Performance and improvements in the MSRCR 

1. One of the tasks under the cooperative agreement NCC-1-258 was to improve the performance 
of the MSRCR on low-end computers. This task was accomplished by a complete redesign 
of the software implementation to allow it to run more efficiently on low-end machines. The 
processing times were reduce from several hours on a machine with 128MB of RAM to under 
an hour. In a similar vein, optimizations in the processing structure resulted in approximately 
three-fold speedups where the memory bottleneck was not an issue. 

2. Related to the improvement in performance were some changes in the actual algorithm. These 
changes were subject of a intellectual property disclosure in May, 1999. The disclosure was 
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jointly filed at the NASA Langley Research Center and the College of William & Mary. The 
relevant document numbers are respectively: LAR 16011-1-CU and WM-003-99. 

Invention — Improvements to the Multi-Scale Retinex with Color Restoration. 

Two innovations have been developed that significantly improve the visual performance of the 
Multi-Scale Retinex with Color Restoration (MSRCR). By generalizing further on the original 
algorithm, optimal solutions have been be obtained that encompass the widest possible range 
of input imagery. The MSRCR is the subject of a previous patent, US Patent ^5,991,456 
granted on November 23, 1999, and numerous publications[l]-[9]. While the MSRCR performs 
very well on diverse digital imagery as a general purpose enhancement method, we have 
developed two new methods that further extend and improve the original MSRCR. These are 

(a) automatic classification of the dynamic range of the incoming image and appropriate 
optimized adjustments in the post-retinex gain and offset, and, 

(b) a post-retinex method for extending the performance for images with large white zones. 
This is especially important in document imaging where white pages with embedded 
color images in addition to the usual text are the norm. 

3. The MSRCR provides the best results when the input image has not been modified in any 
way before processing. However, this cannot be guaranteed for a generally used process. In 
a 1998 conference paer, we investigated how pre-processing affects the final output from the 
MSRCR. 

Z. Rahman, D. J. Jobson, and G. A. Woodell, “Resiliency of the Multiscale Retinex Image 
Enhancement Algorithm, ” Proceedings of the IS&T Sixth Color Imaging conference: Color 
Science, Systems, and Applications, Scottsdale, AZ, (1998). 

Abstract: The multiscale retinex with color restoration (MSRCR) continues to prove it- 
self in extensive testing to be a very versatile automatic image enhancement algorithm that 
simultaneously provides dynamic range compression, color constancy, and color rendition. 
However, issues remain with regard to the resiliency of the MSRCR to different image sources 
and arbitrary image manipulations which may have been applied prior to retinex processing. 
In this paper we define these areas of concern, provide experimental results, and, examine 
the effects of commonly occurring image manipulations on retinex performance. In virtually 
all cases the MSRCR is highly resilient to the effects of both the image-source variations and 
commonly encountered prior image-processing. Significant artifacts are primarily observed 
for the case of selective color channel clipping in large dark zones in an image. These issues 
are of concern in the processing of digital image archives and other applications where there is 
neither control over the image acquisition process, nor knowledge about any processing done 
on the data beforehand. 

4. In addition to the improvements in performance, we extended the use of the MSRCR to 
classification of remotely sensed images. The idea was that since the MSRCR performs 
dynamic range compression and color constancy, it could be used to ameliorate the effects of 
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the atmosphere on the remotely-sensed image resulting in more consistent classification. We 
investigated this idea in two conference publications: 

(a) B. Thompson, Z. Rahman, and S. K. Park, “Retinex Preprocessing for Improved Multi- 
spectral Image Classification, ” SPIE International AeroSense Symposium, Visual Infor- 
mation Processing VIII, (April 1999). 

Abstract: The goal of multi-image classification is to identify and label “similar re- 
gions” within a scene. The ability to correctly classify a multi-image of the scene is 
affected by the ability of the classification process to adequately compensate for the 
effects of atmospheric variations and sensor anomalies. More reliable classification may 
be obtained if the multi-image is pre-processed before classification, so as to reduce the 
adverse effects of image formation. In this paper, we discuss the overall impact on multi- 
spectral image classification when the Retinex image enhancement algorithm is used to 
pre-process multi-spectral image data. The retinex is a multi-purpose image enhance- 
ment algorithm that performs dynamic range compression, reduces the dependence on 
lighting conditions, and generally enhances apparent spatial resolution. The retinex has 
been successfully applied to the enhancement of many different types of grayscale and 
color images. We show in this paper that retinex pre-processing improves the spatial 
structure of multi-spectral images and thus enables better within-class variations than 
would otherwise be obtained with the original images. We show for a series of multi- 
spectral images obtained with diffuse and direct lighting that the class spectral signatures 
vary substantially with the lighting conditions without retinex pre-processing. Whereas 
multi-dimensional clustering without pre-processing produced one-class homogeneous re- 
gions, the classification on the pre-processed data produced multi-class non-homogeneous 
clusters. This lack of homogeneity is easily explained by the different nitrogen and wa- 
ter concentrations in the clusters: the pre-processed being closer to ground truth. The 
principle advantage that the retinex offers is that the classifications from the retinex pre- 
processed images for different lighting conditions look remarkably “similar”, and thus 
more consistent, whereas classifications from the original images, without pre-processing, 
are much less similar. 

(b) B. Thompson, Z. Rahman, and S. K. Park, “A multiscale retinex for improved multispec- 
tral image classification, ” SPIE International AeroSense Symposium, Visual Information 
Processing IX, (April 2000). 

Abstract: Image preprocessing is useful in helping to identify “spectral response pat- 
terns” for certain types of image classification problems. The common artifacts in re- 
motely sensed images are caused by the blurring due to the optics of the image gath- 
ering device, illumination variations, and the radiative transfer of the atmosphere. The 
Multi-Scale Retinex (MSR) image enhancement algorithm that provides dynamic range 
compression, reduced dependence on lighting conditions, and improved (perceived) spa- 
tial resolution has proven to be an effective tool in the correction of image degradations 
such as those in remote sensing images. In this paper, we measure the improvement 
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in classification accuracy due to the application of the MSR algorithm. We use sim- 
ulated images generated with different scene irradiance and with known ground truth 
data. The simulation results show that, despite the degree of image degradation due to 
changes in atmospheric irradiance, classification error can be substantially reduced by 
preprocessing the image data with the MSR. Furthermore we show that, similar to the 
results achieved in previous work, the classification results obtained from the MSR pre- 
processed images for various scene irradiance are more similar to each other than are the 
classification results for the original unprocessed images. This is evident in the observed 
visual quality of the MSR enhanced images even before classification is performed, and 
in the difference images obtained by comparing image data under different irradiance 
conditions. We conclude that the application of the MSR algorithm results in improved 
visual quality and increased spatial variation of multispectral images that is also optimal 
for certain types of multispectral image classification. 


3 Theoretical assessment of non-linear retinex image enhance- 
ment 

The work we have cited above is experimental in both development and in assessment. In order 
to develop a qualitative understanding of the algorithm, we have been working on an information 
theoretic assessment of the non-linear retinex- like processes. A brief description of the work follows; 

• N. Halyo, Z. Rahman and S. K. Park, “Information content in non-linearly processed digital 
images, ” under preparation. 

Introduction: Digital images, whether obtained by electro-optical or by purely optical in- 
strumentation, provide information about the scene that the depict. The amount of infor- 
mation (about the original scene) contained in the processed image is a quantity of interest 
in the design of the image-gathering instrumentation ris well as in the development of digital 
processing algorithms. The rate at which the information in the scene is transferred to the 
processed digital image is a good indicator of the efficiency of the image gathering system 
and the processing performed. 

The information rate has been studied so far only for the special case of signals with Gaussian 
(Normal) power spectral densities (PSDs) in linear image-gathering and either linear [10, 11, 
12] or linearized [13] digital processing systems. In this paper, we will investigate the rate 
of information transferred from the scene to the digital image for systems using non-linear 
processing and containing signals which do not necessarily have Gaussian PSDs. In particular, 
we will investigate the effects of Retinex and Local Normalization (LN) processing. While a 
closed form solution has been available for linearly processed Gaussian signals[14, 15], this 
solution is not valid when either the signals have non-Gaussian characteristics or when the 
processing involved is non-linear. We will investigate and develop an analytical form for the 
information rate for a specified class of non-Gaussian signals and for the Retinex and LN 
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processing. 


Human perception of images depends more on the local characteristics of the image rather 
that the absolute magnitude of the image signals. Thus, the “eye-brain” or “retina-cortex” 
(retinex) system perceives image characteristics relative to the local (spatially weighted) av- 
erage conditions. For example, the same gray level value will appear brighter when it is in a 
dark neighborhood than when it is in a bright neighborhood. 

The main concept in LN processing is to obtain a weighted average of the digital image in a 
local neighborhood around each pixel and normalize that pixel value relative to this weighted 
average. Accordingly, the same gray level value will now be “brighter” if it is located in a 
shadow, i.e. low local average, and will be “darker” if it is located in a bright area, i.e. high 
local average.* In addition, in order to obtain the information rate with LN processing we 
concentrate on that portion of the image signal that describes the detail or high spatial- 
frequency information, instead of analyzing that part of the signal which is often due to 
variations on scene illumination rather than actual scene characteristics. 

In considering the LN processing it is of interest to know how the rate of information varies 
with the application of the non-linear processing. Is the rate of information between the 
scene and the final digital image unchanged (or higher or lower) if the LN process is used 
instead of the well-known linear processes that are traditionally used? Is there an effect on the 
information rate if non-Gaussian scene characteristics are used as opposed to the traditional 
Gaussian PSDs? Whereas in previous studies we have assumed that the radiance signal is 
Gaussian in order to use the well-known results for the rate of information, it is clear that the 
radiance cannot have negative values and therefore cannot be Gaussian for that reason and 
possibly others. While the Gaussian assumption might well be an adequate approximation, 
it is of interest to find out if that is indeed so. Of course, we would like to obtain a relatively 
simple formula for the rate of information which can then be used to perform quantitative 
analysis similar to those done for the traditional Gaussian case. In the next sections, we 
will investigate and answer all the questions posed above and develop a new formula for 
computing the rate of information for LN processing of a class of signals that is not restricted 
to Gaussian. We will also investigate the impact of the various system parameters, such as 
those used to describe the image-gathering device, on the rate of information. 
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